Prompt Generator for Stable Diffusion
Contents
Prompt Generator for Stable Diffusion#
Note
Install ekorpkit package first.
Set logging level to Warning, if you don’t want to see verbose logging.
If you run this notebook in Colab, set Hardware accelerator to GPU.
%%capture
!pip install ekorpkit[art]
exit()
Preparing the environment#
%config InlineBackend.figure_format='retina'
%load_ext autotime
from ekorpkit import eKonf
eKonf.setLogger("WARNING")
eKonf.set_cuda(device=0)
print("version:", eKonf.__version__)
is_colab = eKonf.is_colab()
print("is colab?", is_colab)
if is_colab:
eKonf.mount_google_drive()
project_dir = eKonf.set_workspace(workspace="/content/drive/MyDrive/workspace/", project="ekorpkit-book")
print("project_dir:", project_dir)
INFO:ekorpkit.base:Setting cuda device to ['A100-SXM4-40GB (id:0)']
INFO:ekorpkit.base:Setting EKORPKIT_WORKSPACE_ROOT to /content/drive/MyDrive/workspace/
INFO:ekorpkit.base:Setting EKORPKIT_PROJECT to ekorpkit-book
INFO:ekorpkit.base:Loaded .env from /workspace/projects/ekorpkit-book/config/.env
version: 0.1.40.post0.dev8
is colab? False
project_dir: /content/drive/MyDrive/workspace/projects/ekorpkit-book
time: 3.27 s (started: 2022-11-10 07:42:57 +00:00)
Load a Generator and Generate Prompts#
To download a certain dataset or model checkpoint, you may need to provide a HuggingFace API token. You can get one from here.
import os
# Set HuggingFace API token
os.environ["HF_USER_ACCESS_TOKEN"] = "YOUR_TOKEN"
from ekorpkit.models.art.prompt import PromptGenerator
pgen = PromptGenerator(verbose=False)
2022-11-10 07:43:16.322453: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
time: 6.94 s (started: 2022-11-10 07:43:15 +00:00)
Loading a model#
Before loading a model, you need to train a model first. To train a model, refer to the Training a Generator section.
# ekorpkit/stable-prompts
pgen.load_model(model_name="ekorpkit/stable-prompts")
time: 4.97 s (started: 2022-11-10 07:43:28 +00:00)
Generating prompts#
You can generate prompts using the generate_prompts function. The function takes the following arguments:
prompt: The prompt to be used for generating prompts. IfNone, the prompt will be generated automatically.num_prompts_to_generate: The number of prompts to be generated.generate_images: Whether to generate images for the prompts.num_samples: The number of images to be generated for each prompt.For other arguments, refer to the following code.
eKonf.print(pgen.generate_config)
{'check_start_token': True,
'max_prompt_length': 50,
'min_prompt_length': 30,
'num_prompts_to_generate': 10,
'prompt': None,
'temperature': 1.2,
'top_k': 70,
'top_p': 0.9}
time: 1.18 ms (started: 2022-11-10 07:43:34 +00:00)
prompts = pgen.generate_prompts(
prompt="people looking out a lonely city street",
num_prompts_to_generate=3,
generate_images=True,
num_samples=2,
)
Prompt: people looking out a lonely city street in the snow, concept art portrait of a blonde anime boy with blond hair, high detail, digital art, by greg rutkowski
Prompt: people looking out a lonely city street, artgerm and greg rutkowski and alphonse mucha, a single glowing white male wolf with blue fur, fur, wolf face, ultra realistic, concept art, intricate details
Prompt: people looking out a lonely city street, trending on artstation by Artgerm and Greg Rutkowski and Alphonse Mucha, 8k, HD
time: 1min 6s (started: 2022-11-10 07:49:23 +00:00)
Generating images for prompts#
results = pgen.generate_images(
prompts=prompts,
num_samples=3,
)
Prompt: people looking out a lonely city street corner. cityscape. By Greg Rutkowski, Ilya Kuvshinov, WLOP, Stanley Artgerm Lau, Ruan Jia and Fenghua Zhong, trending on ArtStation
Prompt: people looking out a lonely city street below a lake on a white background, artstation, concept art, matte painting by craig mullins and Anato Finnstark, high detail, volumetric lighting, octane render, 4K resolution
Prompt: people looking out a lonely city street at sunset, fantasy art by greg rutkowski and craig mullins, trending on artstation
time: 1min 36s (started: 2022-11-10 07:46:05 +00:00)
Generating images for one prompt#
results = pgen.imagine(
text_prompts=prompts[0],
num_samples=6,
num_inference_steps=150,
guidance_scale=10,
)
Prompt: people looking out a lonely city street in the snow, concept art portrait of a blonde anime boy with blond hair, high detail, digital art, by greg rutkowski
time: 1min 30s (started: 2022-11-10 07:51:57 +00:00)
Training a Generator#
Preparing a dataset#
You can use any dataset you want. However, the dataset should be in the format of HuggingFace Datasets.
Using a dataset from HuggingFace Hub
dataset = pgen.load_dataset("Gustavosta/Stable-Diffusion-Prompts")
dataset
WARNING:datasets.builder:Using custom data configuration Gustavosta--Stable-Diffusion-Prompts-d22aeec0ba2a9fdb
WARNING:datasets.builder:Reusing dataset parquet (/content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/cache/Gustavosta___parquet/Gustavosta--Stable-Diffusion-Prompts-d22aeec0ba2a9fdb/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec)
DatasetDict({
train: Dataset({
features: ['Prompt'],
num_rows: 73718
})
test: Dataset({
features: ['Prompt'],
num_rows: 8192
})
})
time: 4.64 s (started: 2022-11-10 07:13:51 +00:00)
pgen.trainer_config.num_train_epochs = 2
pgen.train(
model_name="ekorpkit/stable-prompts",
dataset=dataset,
prompt_column="Prompt",
split="train",
)
PyTorch: setting up devices
loading configuration file https://huggingface.co/distilgpt2/resolve/main/config.json from cache at /content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/cache/f985248d2791fcff97732e4ee263617adec1edb5429a2b8421734c6d14e39bee.422318838d1ec4e061efb4ea29671cb2a044e244dc69229682bebd7cacc81631
Model config GPT2Config {
"_name_or_path": "distilgpt2",
"_num_labels": 1,
"activation_function": "gelu_new",
"architectures": [
"GPT2LMHeadModel"
],
"attn_pdrop": 0.1,
"bos_token_id": 50256,
"embd_pdrop": 0.1,
"eos_token_id": 50256,
"id2label": {
"0": "LABEL_0"
},
"initializer_range": 0.02,
"label2id": {
"LABEL_0": 0
},
"layer_norm_epsilon": 1e-05,
"model_type": "gpt2",
"n_ctx": 1024,
"n_embd": 768,
"n_head": 12,
"n_inner": null,
"n_layer": 6,
"n_positions": 1024,
"reorder_and_upcast_attn": false,
"resid_pdrop": 0.1,
"scale_attn_by_inverse_layer_idx": false,
"scale_attn_weights": true,
"summary_activation": null,
"summary_first_dropout": 0.1,
"summary_proj_to_labels": true,
"summary_type": "cls_index",
"summary_use_proj": true,
"task_specific_params": {
"text-generation": {
"do_sample": true,
"max_length": 50
}
},
"transformers_version": "4.21.2",
"use_cache": true,
"vocab_size": 50257
}
loading weights file https://huggingface.co/distilgpt2/resolve/main/pytorch_model.bin from cache at /content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/cache/43a212e83e76bcb07f45be584cf100676bdbbbe9c13f9e5c1c050049143a832f.a83d881ec4d624fd4b5826dd026e315246c48c67504ff91c0500570e291a54ba
All model checkpoint weights were used when initializing GPT2LMHeadModel.
All the weights of GPT2LMHeadModel were initialized from the model checkpoint at distilgpt2.
If your task is similar to the task the model of the checkpoint was trained on, you can already use GPT2LMHeadModel for predictions without further training.
***** Running training *****
Num examples = 18847
Num Epochs = 2
Instantaneous batch size per device = 1
Total train batch size (w. parallel, distributed & accumulation) = 8
Gradient Accumulation steps = 1
Total optimization steps = 4712
Automatic Weights & Biases logging enabled, to disable set os.environ["WANDB_DISABLED"] = "true"
/workspace/projects/ekorpkit-book/ekorpkit-book/docs/lectures/aiart/wandb/run-20221110_071621-38wtdfq3| Step | Training Loss |
|---|---|
| 200 | 3.800000 |
| 400 | 2.698200 |
| 600 | 2.500200 |
| 800 | 2.353500 |
| 1000 | 2.253200 |
| 1200 | 2.165000 |
| 1400 | 2.111500 |
| 1600 | 2.049800 |
| 1800 | 2.008200 |
| 2000 | 1.974800 |
| 2200 | 1.924800 |
| 2400 | 1.912100 |
| 2600 | 1.869900 |
| 2800 | 1.851300 |
| 3000 | 1.832200 |
| 3200 | 1.807700 |
| 3400 | 1.796100 |
| 3600 | 1.775500 |
| 3800 | 1.767700 |
| 4000 | 1.761300 |
| 4200 | 1.749900 |
| 4400 | 1.735000 |
| 4600 | 1.750500 |
Training completed. Do not forget to share your model on huggingface.co/models =)
Run history:
| train/epoch | ▁▁▂▂▂▃▃▃▃▄▄▄▅▅▅▆▆▆▇▇▇███ |
| train/global_step | ▁▁▂▂▂▃▃▃▃▄▄▄▅▅▅▆▆▆▇▇▇███ |
| train/learning_rate | ██▇▇▇▆▆▆▅▅▅▄▄▄▄▃▃▃▂▂▂▁▁ |
| train/loss | █▄▄▃▃▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁ |
| train/total_flos | ▁ |
| train/train_loss | ▁ |
| train/train_runtime | ▁ |
| train/train_samples_per_second | ▁ |
| train/train_steps_per_second | ▁ |
Run summary:
| train/epoch | 2.0 |
| train/global_step | 4712 |
| train/learning_rate | 0.0 |
| train/loss | 1.7505 |
| train/total_flos | 9849319716814848.0 |
| train/train_loss | 2.05561 |
| train/train_runtime | 606.0063 |
| train/train_samples_per_second | 62.201 |
| train/train_steps_per_second | 7.775 |
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
./wandb/run-20221110_071621-38wtdfq3/logsConfiguration saved in /content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/models/ekorpkit/stable-prompts/config.json
Model weights saved in /content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/models/ekorpkit/stable-prompts/pytorch_model.bin
time: 11min 13s (started: 2022-11-10 07:15:21 +00:00)
Using a dataset from a text file
prompt_uri = "https://raw.githubusercontent.com/entelecheia/ekorpkit-book/main/assets/data/prompt_parrot.txt"
dataset = pgen.load_dataset("text", data_files=prompt_uri)
dataset
WARNING:datasets.builder:Using custom data configuration default-681f997af4470be8
WARNING:datasets.builder:Reusing dataset text (/content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/cache/text/default-681f997af4470be8/0.0.0/21a506d1b2b34316b1e82d0bd79066905d846e5d7e619823c0dd338d6f1fa6ad)
DatasetDict({
train: Dataset({
features: ['text'],
num_rows: 185
})
})
time: 1.29 s (started: 2022-11-10 07:11:28 +00:00)
eKonf.print(pgen.train_config)
eKonf.print(pgen.trainer_config)
{'pretrained_model_name_or_path': 'distilgpt2',
'prompt_column': 'Prompt',
'split': 'train'}
{'logging_steps': 200,
'num_train_epochs': 1,
'overwrite_output_dir': True,
'per_device_train_batch_size': 1,
'prediction_loss_only': True,
'report_to': 'wandb',
'run_name': 'ekorpkit/stable-prompts',
'save_steps': 0,
'seed': None}
time: 3.54 ms (started: 2022-11-10 07:11:32 +00:00)
pgen.trainer_config.num_train_epochs = 2
pgen.train(model_name="ekorpkit/prompt_parrot", dataset=dataset, prompt_column="text")
Token indices sequence length is longer than the specified maximum sequence length for this model (22608 > 1024). Running this sequence through the model will result in indexing errors
***** Running training *****
Num examples = 22
Num Epochs = 2
Instantaneous batch size per device = 1
Total train batch size (w. parallel, distributed & accumulation) = 8
Gradient Accumulation steps = 1
Total optimization steps = 6
Automatic Weights & Biases logging enabled, to disable set os.environ["WANDB_DISABLED"] = "true"
ERROR:wandb.jupyter:Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
wandb: Currently logged in as: entelecheia. Use `wandb login --relogin` to force relogin
/workspace/projects/ekorpkit-book/ekorpkit-book/docs/lectures/aiart/wandb/run-20221110_071154-29lg6d7y| Step | Training Loss |
|---|
Training completed. Do not forget to share your model on huggingface.co/models =)
Run history:
| train/epoch | ▁ |
| train/global_step | ▁ |
| train/total_flos | ▁ |
| train/train_loss | ▁ |
| train/train_runtime | ▁ |
| train/train_samples_per_second | ▁ |
| train/train_steps_per_second | ▁ |
Run summary:
| train/epoch | 2.0 |
| train/global_step | 6 |
| train/total_flos | 11497057026048.0 |
| train/train_loss | 17.0794 |
| train/train_runtime | 56.6702 |
| train/train_samples_per_second | 0.776 |
| train/train_steps_per_second | 0.106 |
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
./wandb/run-20221110_071154-29lg6d7y/logsConfiguration saved in /content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/models/ekorpkit/prompt_parrot/config.json
Model weights saved in /content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/models/ekorpkit/prompt_parrot/pytorch_model.bin
time: 1min 18s (started: 2022-11-10 07:11:37 +00:00)